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ABSTRACT 

toward, the Development of 
An Ethnographic Model for Program Evaluation 

by 

Carl K. Chafin, Ph.D. 
Virginia State University 

A problem^ endemic to program evaluation, centering around the in- 
ability of evaluations to -produce meaningful information, is outlined. A 
A genera'' strategy to be utilized in responding to the problem, the ethno- 
graphic approach, is presented. 

The discussion of an ethnographic evaluation model focuses upon: the 
role of the participant observer, establishing oneself as a participant 
observer evaluation, and the handling of sensitive information. The ex- 
ploratory activities out of which came the initial issues to be investigated 
are described, along with some observations about the process of delimiting 
the investigation, andoof initiating "personal*' relationships. Discussion 
is interspersed throughout the paper, focusing upon the methods used to 
collect data, and the line of investigation pursued during the study. 
Finally, the advantages of the ethnographic approach to evaluation as 
compared to the experimental approach are discussed. 
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TOWARD THE DEVEbOPMENT OF AN ETHNOGRAPHIC MODEL FOR PROGRAM EVALUATION 

Program evaltiatibri has been enmeshed in a fundamental controversy 
throughout its relatively short existence as a discipline. The conflict 
has been the result of the seemingly irreconciable differences created by 
its heritage and its present focus of inquiry. The predominant ancestors 
from which evaluation developed are the experimental research tradition 
arid the psychological measurement movement, both of which have served as 
the foundation for educational research throughout the twentieth century. 

The most popular design for evaluation istudies today (for both prac- 
titioners and sponsors of evaluation) is a direct descendant of Ronald 
Fisher's work on the design and conduct of experiments in the late nine- 
teenth and early twentieth century. Fisher pioneered efforts in the de- 
sign of experiments and the statistical treatment of data for the purpose 
of drawing inferences as to the causal relationship between variables. 
The concept of randomization, the techniques of analysis of variance and 
analysis of eovarianee, and the use of the null hypothesis were all uti- 
lized by Fisher in his watershed experiments in the fields of biology 
and agriculture. Through the work of FisHer, the practice of observing 
phenomena in order to determine cause-effect relationships assumed a 
precise, rigoubus, and methodical form. Practitioners in the field of 
evaluation have, over the years, concentrated their efforts upon refin- 
irig the Fisher model arid using it to assess the impact of educational 
innovations and other programs designed to ameloriate social problems. 

eoupled with this type of design for evaluation studies was the par- 
ticular technique of collecting data on participants in a program via in- 
struments designed to measure the specific trait(s) or eharacteristic(s) 
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that programs were attempting to change. The development of these 
types of data collection instraments was largely stimulated by 
the wbrks of dames Cattell, Edward Thorndike, and Lewis Terman, 
during the first part of the twentieth century (Thorndike, 1951 ^ p. 5). 
The emphasis in the educational measurement movement was upon the use 
of paper and pencil tests, administered to an individual or groups of 
individuals in order to produce information about such things as intel- 
lectual ability, personality, and performance. The flavor of program e-^ 
valuation methodology which grew out of this educational research tradi- 
tion reflects very strongly the experimental design and testing influence. 

Problems began to arise, however, when researchers were called upon in 
the 1960' s to determine the effectiveness of educational and social reform 
programs that were being funded primarily by the federal government. The 
initial attempts to evaluate these programs involved using what were at 
the time the best methods available; the traditional research design and 
measurement strategies. This type of research, hoWever, is heavily depen- 
dent upon a setting that can be controlled by the researcher; it demands 
a very precise manipulation of environmental variables; and the program* 
both internally and in relation to its larger context, must approach a 
homeostatic state. The California Gold Rush probably more closely re- 
flected these conditions than did the social intervention programs of 
the 1960*s, and understandably so. What was being tried in these pro- 
grams was unique. The society was diverting huge sums of money into the 
development of programs designed to cure its many ailments. In reflect- 
ing back over a decade since these first attempts, it is easy to conclude 
that there was a naive optimism prevalent which engendered the belief 
that not only would these programs have some impact, but they would sta- 
balize after a short period of time and thus enable this impact to be 
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measured. Unfortunately, this was not the case and as the 6G's wore on* 
the optimism faded • 

The fact that evaluation studies carried out in the traditional man- 
ner failed to discover any significant program impact stimulated a varie- 
ty of explanations. The ones most often heard were: the idea was no 
good right from the start because our lot in life is determined either 
by how hard we try to help ourselves, or by our ability; the programs 
were not given enough time and money to work their magic; the programs 
were poorly managed and finally, they actually did accomplish something, 
but the evaluators were collecting the wrong kind of data and could not 
ascertain the true impact of the programs. Irrespective of which of the 
above explanations has the most validity, the last one had the greatest 
impact on the practice of evaluation. 

Edmond Gordon typified the prevalent attitude when writing in 1970 
on the evaluation of compensatory education programs: 

"Quite apart from the problems related to the conditions under which pro- 
grams were initiated and conducted are the problems of evaluative research 
in general. Here* one often finds a low level of expertise and inadequate- 
ly developed methods. The best educational research scientists often 
choose to work With basic problems in areas such as child development, 
learning, linguistics, rather than with evaluative research. Evaluative 
arid field research have only recently gained in respect and demand among 
educators arid the public. Consequently, high demand has been suddenly 
created in a field with insufficient expertise. Although many good re- 
search scieritists were drawn into evaluation, they could not readily trans- 
fer their research competence to the new situation. Indeed, given their 
experience in controlled laboratory settings, the problems of evaluative 
and field research may have been more difficult for them than for some 



less experienced investigators" (1970, p. 2). 

Robert Rippey^ in a book entitled Studies in Transactional Evalua- 
tion ^ is likewise very skeptical about the value of evaluations: 
"Most evaluations to date have been useless. Formative evaluations usu~ 
ally come too late often after a program has lost, or retained, its fund- 
ing. Summative evaluations, despite their intent, have been inconclusive 
At the moment* there seems to be no evidence that evaluation, although 
the law of the land, contributes anything to educational practice other 
than headaches for the Innovators, and depressing articles for journals 
devoted to evaluation" (1973, p. 9). 

Egon Guba suggests that the widespread dissatisfaction with evalua- 
tion is largely due to methodological inadequacies: 

"...many agree that evaluation has not been as fruitful as had been hoped 
or expected. Useful evaluation information is not often produced; and 
even when it is* decision-makers and policy formulators sometimes see fit 
to disregard it. While there are undoubtedly many reasons for this state 
of affairs* a major reason seems to be the lack of methodology uniquely 
suited to evaluation's needs" (.1978, p.l). 

This paper is offered in the belief that program evaluation is pre- 
sently in a predicament analogous to that of the child who was taught to 
swim by being thrown in the creek and instructed to "sink or swim." bike- 
wise, we as evaluators are forced to develop our techniques while we si- 
multaneously struggle to keep our heads above water. The dilemma is fur- 
ther cdmpdunded by the fact that evaluators are expected to not only make 
some judgement as to a program's worth, but also to provide information 
to program managers, and to conduct research. As practitione.^s in a de- 
veloping discipline, we are not afforded the luxury of simply practicing 
our trade. We must distill information from every field work experience 



that wi11 contribute to bur efforts to create conceputally sound models 
and viable methods for eonductirig evaluation studies; the intent of this 
paper is to make such a contribution. 

Overview of the Model 

The study from which the model grew was an evaluation of the Alter- 
nate Community Service Program (ASCP), a volunteer work program for ju- 
venile offenders who have committed minor violations of the law, AGSP 
operates in Charlottesville, Virginia and is envisioned as a model pro- 
gram, respon^.iJig to a need within the juvenile justice system for al- 
ternate forms of disposition that can be made available to officers of 
the court to use when such time-honored forms of disposition as dismis- 
sal , probation, and incarceration are deemed inappropriate. 

The model is summarized in Figure 1. the basic approach is "utili- 
zation-focused," for it is "...aimed at increasing the likelihood that 
evaluation input will be substantial^ meaningful, and relevant" (Patton, 
1978, p. 34). It responds to Patton' s definition of "utilization" as 
being "...not something that suddenly and concretely occurs at some one 
distinct moment in time. Rather, utilization is a diffuse and gradual 
process of reducing decision-maker uncertainty within an existing so- 
cial context" (Ibid, p. 34). the investigation proceeds from "known to 
unknown" in an "expansive/intensive" manner. Any inquiry into a speci- 
fic issue follows a pattern of: collecting information to describe the 
issue, reflecting on the information, and offering an interpretation 
about the issue. From the methodological point of view the investigation 
unfolds through a pattern of intensive interaction between the program 
and the evaluator. In my study, this interaction was primarily between 
myself and two key informants. Related to the pattern of personal in- 
teraction IS the element of continuous feedback of information, at the 



end of a cycle of deplction-reflectidn-ihterpretation, relative to a giv- 
en issue. This feedback is typically informal in that it is not issued 
as a written report, but is shared in conversation. This informal re- 
porting was balanced in the study by the preparation and delivery of 
two formal documents to the program director. These were the "program 
description," and the "final report." 

Insert Figure 1 About Here 

The result of the process of questioning and requestioning people 
IS the accumulation of data that, when complemented by observational data 
and written records, can be used to construct a descriptive picture of 
the program that moves beyond mere depiction to include reflection and 
interpretation. Throughout the course of an investigation ^ the three - 
depiction, reflection, and interpretation - must be intertwined. Inter- 
pretation begins when one first arrives on the scene; the attempt to ac- 
curately depict the setting continues until final departure. The pro- 
gression is from tentative small seal? doscriptidns which subsume de- 
piction, reflection, and interpretation^ to refined descriptions that 
are more comprehensive and less tentative. Clifford Geertz, in The In- 
terpretation of Cultures ^ describes the process as follows: "Previously 
discovered facts are mobilized^ previously developed concepts used, pre- 
viously formulated hypotheses tried but; but the movement is not from al- 
ready proven theorems to newly proven ones^ it is from an awkward fumbling 
for the most elementary understanding to a supported claim that one has 
achieved that and surpassed it" (:1978, p. 25). 

In essence, the approach is characterized by ii Utilitarian perspec- 
tive and an interactive relationship between evaluator and program, the 
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investigation proceeds on the basis of a thorough tinderstandirig of the 
program's intent, and design, to explore critical issues through an ex- 
pansive/intensive inquiry and to offer meaningful interpretations to 
the program staff, sponsors, and clientele. 

The discussion below focuses primarily upon the process component 
of the model, particularly the methods for establishing a role and gain- 
ing an understanding of a program, and for expanding and intensifying 
the inquiry. Because ethnographic research generally uncovers "sensi- 
tive" information, the problem of how to handle this information will 
be examined. Finally, the advantages of ethnographic over experimental 
research* for program evaluation, are discussed. 

Es^tabl-i^-Rale /Gain thorough Understanding 
The role of the participant observer can be likened to the role of 
a parent attempting to guide a child through life. The development of 
a child involves an almost continuous redefinition of "reality" on the 
basis of new experiences and knowledge gained each succeeding day. The 
child's behavior can be understood as an attempt to respond to, or in- 
teract with, reality as he or she understands it at that moment. This 
interaction causes the child's perception of reality to be altered or 
refined. A new and tentative understanding of reality is achieved that 
serves as an intellectual oasis until the child is refreshed and ready 
to resume his explorations. Ever present, the parent attempts to pre- 
serve potent symbols of cht younger days while providing insight and di- 
rection as the child grown older. At no time does the parent's "reality" 
become one and the same with the "chi1d*s," yet the parent's perspective 
allows him to "observe" the child as he incessantly processes information 
derived from new experiences, reshapes "reality," and refines his under- 
standing of it. While "observing," and attempting to preserve elements 
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of the child's experience, the parent exerts a guiding influericei he can- 
not live the ehild's life for him, and in that sense he is "objectified" j 
he can profoundly influence the ehild's development, and in that sense he 
is "involved." 

The participant observer as an evaluator can achieve a similar rela- 
tionship with a social action program. To illustrate, in an alternative 
school for dropouts which I helped to evaluate, the original proposal for 
the project emphasized the importance of getting parents of students ac- 
tively involved in the school. While students were being recruited for 
the program, it became obvious to the staff that involving parents was 
not a realistic objective because students were adamantly opposed to it. 
Many had moved out of their homes becasue they were being beaten, humil- 
iated, or abused in other ways. Thus, the decision was made to eliminate 
plans for parental involvement. In a related problem, the same project 
intended to have teachers provide counseling services to the students. 
However, a number of students came to teachers for help with problems 
(rabstly family-related) that were so severe in nature, teachers did not 
feel qualified to respond. 

In both of these examples, the project staff was forced to respond to 
a "reality" that was different from what they had anticipated. The form 
of the project was altered by the staff as they attempted to respond to 
this changing "reality." As part of their "involvement" in this program, 
the evaluation team assisted the staff in coping with a "reality" that 
changed as a result of new experiences and information, by helping them 
to develop new strategies for dealing with students' personal problems. 
From the perspective of "objectivity" or "hon-invbl vement" the evaluatbrs 
documented the existence of these problems, the extent to which they in- 
fluenced the program, and the changes that were made by the staff in 
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responding to them. We had established ourselves as being "insiders" to 
the degree that we were asked to take part in the staff deliberations rel- 
ative to these problems, but we retained the "outsider" perspective such 
that we were able to view the problems as they arose, the staff's response 
to themj and record this as part of the documentation of the program's de- 
velopment. The participant observer/ evaluator is able to collect some 
data and be of some value becasue he is perceived as being "like" an in- 
sider-; but he is able to collect other data and reflect on all data dif- 
ferently because he is an outsider. 

Th# RoT^g- of -I ns44er/0yteidar 
Pel to develops the insider /outsider role further from the point of 
view of an anthropologist in the field: 

"Some people have written as if the ethnographer gains access to local 
private information to the degree that he becomes identified as a local 
- an insider. This is, Df course, very important - up to a point. Beyond 
that point, the fieldworker is privy to significant information because 
he is an outsider ^ someone who is different from every other member of 
the cornmunity. He is different because his core prestige ultimately rests 
on membership in another society; he is neutral in the local competitive 
scene... he provides the possibility for social interaction in which the 
rules are suspended to a certain extent. Friendship and social affilia- 
tion can be obtained from him with relatively little social risk, because 
he is not competing for prestige in the social arena" (Pelto, 1980, p-220). 

The nature of the personal relationships that one must form in estab- 
lishing oneself in the insider /outsider role is not always clearly under- 
stood by the prospective participant observer. It is naive to assume that 
the personal relationships the researcher needs to develop with the "lo- 
cals" will assume their proper form naturally. The researcher should be 



fully aware of the type of relationship that needs to be created. On the 
surface, the researcher must become "friendly'* with the people, display- 
ing an interest in what they are doing and listening to them when they 
talk. However, in order to extract information from people, instead of 
simply acting as a passive recipient of information ^ the researcher must 
attempt to create a relationship that moves beyond "friendly." 
ftelati^nsh4p-Based on Intimate Eontent 

Raymond Gold, in an article entitled "Roles in Sociological Field 
Observations," describes the type of relationship the participant observ- 
er attempts to create, and he bases his explanation on George Simmel's 
distinction between "intimate form" and "intimate content." He cites 
the following passage from The Sociology of George Simmel ^ edited by 
Kurt Wolff: 

"...intimacy is not based on the aontent of the relationship. . .certain 
external situations or moods may move us to make very personal statements 
and confessions, usually reserved for our closest friends only* to rela- 
tively strange people. But in such eases we nevertheless feel that this 
* intimate' content does not yet make the relation an intimate one... That 
'intimate' content, although we have perhaps never revealed it befdre* 
and thus limit it entirely to this particular relationship^ does never- 
theless not become the basis of its form, and thus leaves it outside the 
sphere of intimacy" (Wolff, 1964, p. 127). 

Simmel further develops his concept of "intimacy" as follows: 
"In probably each relation, there is a mixture of ingredients that its 
participants contribute to it alone and to no other^ and of other ingre- 
dients that are riot characteristic of it exclusively but in the same or 
similar fashion are shared by its members with other persons as well. 
The peculiar color of intimacy exists if the ingredients of the first 



type, or more briefly i if the "internaV' side of the relation is felt to 
be essential; if its whole affective structure is based on what each of 
the two participants gives or shows only to the other person and to no- 
body else" (Ibid, p. 126). 

To be effective, the participant observer must carefully orchestrate 
the development of relationships so that they facilitate the exchange of 
"intimate content," but do not actually achieve a state of "intimate form-" 
The fully articulated role of the participant bbserver goes far beyond be- 
ing just a friendly fellow that people enjoy talking to. A great deal of 
purposive and subtle molding and massaging of personal relationships must 
take place if the research is to be successfully carried out. Additional- 
ly, for the researchers attempting to do an evaluation as an ethnographic 
study, a unique problem arises relative to the expectations of the staff 
and sponsors of the program which adds a new dimension to the problem of 
establishing a role. 

Providing a Service While Defining the Rdlg 

An anthropologist in the field establishes a relationship with the 
people being studied that allows them to be observed oh a daily basis, 
and to be questioned about what it is they are doing and why they are 
doing it. The relationship that allows for meaningful inquiries to be 
made is one in which the anthropologist's presence is at least tolerated, 
if not openly appreciated. The anthropologist must devote whatever time 
is necessary to gain the right to be present and the ultimate research 
objectives are dependent upon the creation of this acceptable role. 

The evaluator encounters a similar problem in trying to become a 
participant observer in a program; he or she must acquire the right to 
be present as daily patterns of activity unfold. However^ this problem 
takes on an added dimension becasue the clients of the evaluation expect 
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the evaluator to perform some type of service * and in most cases they are 
paying for the service. The anthropologist does riot have as the focal 
point of his study the provision of a service to a community. Thus, 
the anthropologist can afford to bide his or her time in order to ensure 
that the proper role is created; whereas the evaluator is riot afforded 
this luxury. An evaluator is supposed to be doing something that is of 
value to either the program staff, the sponsors, or both. As a result* 
the evaluator must demonstrate early in the association with a program 
that a service can be provided to those responsible for the program 

The anthropologist has in his or her bag of methodological tricks 
certain pursuits chat can be engaged in while a role is defined and de- 
veloping relationships are nurtured; the archetype is the delineation of 
kinship system of the people being studied. Although the study of kin- 
ship systems has long been a focal point of research in anthropology for 
a plethora of reasons, the ones discussed below are the most relevant to 
the problem of how to conduct research while simultaneously defiriing a 
role for oneself in the research setting. The first advantage to inves- 
tigating kin relations is that it is something most everyone, in any cul- 
^ ture, is willing to talk about, even to a semi-stranger. A second as- 
pect of kinship study is that it is a convenient starting point for learri- 
ing the local language. Finally, as the anthropologist begins to illumi- 
nate and understand the web of kin relations and their linguistic refer- 
ents, an insight is gained into aspects of life in that particular cul- 
ture, such as economic relationships; division of labor; the exercise of 
power, authority, and influence; the particularities and peculiarities 
of behavior associated with affinal and consariguineal relations, and so 
Oh. The point is that pursuing questions about kinship enables the an- 
thropologist to use the research activity to establish a foothold in the 
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community. From this base* he or she can proceed to branch out into other 
areas of inquiry that were previously stated in the research proposal, or 
newly discovered in the research setting. 

The participant dbserver/evaluator^ likewise, needs a methodological 
"triefc" that will allow data to be gathered while he or she is becoming 
established in the program. Also^ in response to a concern mentioned pre- 
viously, the program staff needs to be convinced, within a short period of 
time, that the evaluator is capable of helping them and is a worthwhile 
person with whom information should be shared. Additionally, the newly 
arrived evaluator has a "language" or jargon that must be learned in or- 
der to be able to converse freely and meaningfully with the program staff; 
To respond to these concerns the evaluator, like the anthropologist in the 
field, needs to find a subject that people in the program would be willing 
to discuss as they become familiar with^ and begin to understand, the role 
being assumed: and his or her value to them in that role. 

In the evaluation study that forms the basis of this paper, I respond- 
ed to the problems of establishing a role and demdhstrating my value by 
using a technique for developing a "program design" that is part of the 
late Malcolm Prdvus' Discrepancy Evaluation Model (DEM). I had used this 
particular technique countless times as a staff member of a project fund- 
ed by the Bureau of Education for the Handicapped^ designed to teach spe- 
cial educators operating teacher training programs how to do their own 
program evaluations. A program design in the DEM consists of two parts. 
First, the program in question is defined as consisting of a number of 
"functional" components. Each of these functional components is explained 
in terms of: ij its needed inputs^ 2) the interaction that is expected to 
take place among the inputs; simply put* the "process", and 3) the "out- 
puts" that are to be produced as a result of the process. 
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In the project mentioned above, a staff member would sit down with 
participants and assist them, or elecit from them a program design. The 
idea was to proceed from the general to the specific. Gne began by try- 
ing to get an overview of the program; from this, several major compo- 
. nents were defined and each was described in terms of its requisite in- 
puts, its anticipated process of operation, and its expected outcomes, 
dnce each component was described this way at a general level, the next 
step was to take the major components and further define each in terms 
of its functional sub-components. These sub-components were then de- 
scribed in terms of their inputs, processes, and outputs. If done pro- 
perly, the end result was a clear and fairly concise statement regarding 
how a program was designed to operate and what its objectives were. 

The process of eliciting a program design involved getting people 
to talk about their programs, and my experience suggested that this topic 
rivaled kinship as an object of enthusiastic conversation. Furthermore, 
lengthy discussion about a program enables the evaluator to begin famil- 
iarizing himself with that particular jargon. During the period spent 
with the staff formulating the program description, the ^valuator devel- 
ops a comprehensive picture of the program and discovers potentially 
fruitful avenues to pursue during the study. Finally, and most impor- 
tantly in the initial role development stage, the program design process 
culminutes in a tangible product, a program design document, that I have 
found to be useful to program staff: 1) as an aid to program monitor- 
ing, 2) as a vehicle for disseminating "model" programs, and 3) in the 
preparation of proposals. 

By engaging the staff in the development of a program design I im- 
mediately demonstrated my value and, therefore, was able to legitimate 
jry presence. In a sense, I used the development of a program design as 
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a way of buying the time needed to establish myself in the program. 

E xpanding and Intensifying the fnvestiqatibn 
Ghee the program design was completed, the investigation vacillated 
bcrbweeh what could be called "expansive" and "intensive. " The prototype 
ef this Inquiry is the archaelogical excavation. 

Gnce a site has been selected and a grid has been precisely laid out 
over it, digging can begin. • Individual squares are opened up and excava- 
ted. Work progresses along two dimensions: individual squares are exca- 
vated in depth and new squares are opened up. Discoveries in individual 
squares may suggest a direction in which new squares should be opened. 
For example, the discovery in a square of what appears to be a stone foun- 
dation might suggest opening the adjacent squares to attempt to uncover 
more of the foundation. The entire site is opened up on the basis of 
this rule. The archaelogist encounters may deadends, but as discarded 
hypotheses become no more than intellectual backfull, new hypotheses are 
suggested almost as quickly as each succeeding inch of dirt can be re- 
moved. The past is reconstructed as the original squares are dug deeper 
and as additional squares open up greater areas of the site. Expansion 
and intensification complement one another in gradually revealing evi- 
dence of the past life, enabling inferences to be made about it. 

In gathering data to evaluate a program one proceeds similarly. The 
experience gained in developing the program design suggested a direction 
for an inttial investigation. The strategy, as the data collection pro- 
cess unfolds, was to expand the inquiry to gain a fuller understanding of 
the nature and purpose of the program and the social and political context 
within which it exists; and to focus upon and probe intensely specific aspects 



of the program. Like the archaeolgical excavation, intensifying the in- 
vestigation in one area should lead to a broadening of the investigatory 
horizon; and expanding the scope of the inquiry should create new areas 
that merit more intense scrutiny. The result would, hopefully, be ah 
interpretation or evaluation that would be meaningful to the clients be- 
cause it would address itself in a knowledgeable manner to the illumiha- 
tidri of problems with the program vis-a-vis the larger context and to 
problems that may indirectly relate to context, but are more directly 
the result of internal policies and practices. 
Questioning Strategies 

To produce the type of study outlined above, I relied primarily on 
data gathered through my own observations, questions, and through analy- 
sis of the program's written records. Data previously collected, expe- 
rience gained through my continuous involvement in the program, the in- 
cipient interpretations of data^ and simple intuition guided my ques- 
tioning. 

Specific questions fell into the categories of "reportorial " and 
"posing," suggested by Anselem Strauss* et al., in P s V€H4atr T^Id^to^ies^ 
and Institutions , (1964). The types of questions are defined below and 
each is followed by ah example from the actual study: 

"Reportorial" questions focus upon "...the who, what, where, how, 
and why of events." Examples of "reportorial" questions that I asked 
are: "Who is eligible to participate in the program?", "What problem 
is the program responding to?", "Where did the idea for the program orig- 
inate?", "Row are prospective participants selected?" "Posing" questions 
are of four types. 

"The challenge or devil's advocate question. The fieldworker de- 
liberately confronts the respondent with the arguments of opponents. The 
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idea is to elicit rhetorical assertion and thus to round out the respon- 
dent's position by forcing him to respond to challenge;" In reviewing 
the file on participants. I discovered that a number of people were admit- 
ted to the program who did not meet the selection criteria. I asked the 
director, "Why were the criteria enforced in some cases and relaxed in 
others?" He responded that the "criteria" were meant to serve as "guide- 
lines" for those officers of the court sending potential participants to 
the program. In some individual cases, the guidelines were not followed 
because of the unique circumstances Of that particular case. 

"The hspotheiiaat question, the fieldworker poses a number of pos- 
sible occurences - for example. What if Csomeone) did this or that? What 
would happen (if)...?" In the project, participants perform volunteer 
community work in lieu of probation or a fine for committing an offense. 
I asked a staff member, "What would happen if a participant failed to 
complete the volunteer work?" The answer I received was that the case 
would be returned to the court and re-opened. 

"Posing the ideal. There are two variations on this technique. 
First, the respondent can be asked to describe the ideal situation. . .Sec- 
ond, the fieldworker... can assert an ideal to see what response is eli- 
cited." I posed the following question to a staff member, "Ideally, 
would ewry offender whose case satisfied all of the program's selection 
criteria be referred to the program for an interview and possible accep- 
tance?" The response was affirmative. 

"Offering intergvetation or testing propositions on respondents. It 
is sometimes very useful to tell respondents about the propositions that 
one is beginning to pull together. .. If they disagree, they will usually 
volunteer information to counter a proposition, which may lead the field- 
worker into further unanticipated search. If they agree, the tendency is 
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to qualify the proposition: it does not quite meet the case. Again ^ the 
fieldworlcer comes away with additional valuable information'' (Strauss, 
1969, pp.71-72j, I discovered, at one point, that a staff memBer believed 
that the program's selection process discriminated against blacks, yet in 
a private meeting with me, the project director (who was not aware of the 
contrasting belief of the staff member) indicated that he did not believe 
this to be the case. In a meeting with the two of them, I questioned the 
extent to which the program jnight discriminate, and a discussion ensued be- 
tween the two which forced the director to "respond to the challenge" raised 
by the contrasting point of view. (He subsequently decided to look more 
closely at the program's selection process.) 

Handling Sensative Information 

A natural consequence of conducting ethnographic research is that the 
researcher becomes a party to information concerning a wide range of topics. 
A great deal of this information is characterized by "intimate content," as 
defined previously. The fundamental dilemma for the researcher is how to 
handle this information once it has been obtained. 

As there is no lockstep method for collecting data in ethnographic re- 
search, there is, likewise, no pre- fabricated scheme to direct the research- 
er once he has obtained detailed and sensitive information. Yet, the very 
fact this problem is not easily resolved makes a discussion of it all the 
more critical. However before analyzing the problem, I will further define 
it via an illustration from llndeys^taiidijig an^fri^an-^lngdom^^ Bun y^m , by 
John Beattie. He states: 

"Obviously, no responsible anthropologist will betray to the authorities the 
fact, say, that a neighbor has been distilling illicit liquor, or has success- 
fully evaded a tax obligation. But, it is very much a matter of degree. When 
I learned, for example, that a respected neighbor, employed in the local hos- 
pitali was stealing syringes and giving injections with an unsterilized needle 

,21 

ERIC 



ERIC 



to iQcal people for a fee, I felt justified in suggesting to the medical 
authorities that ihereased vigilance might be desirable (and in attempting 
to persuade the amateur physician of the importance of asepsis). But I did 
not feel justified in reporting the matter to the police. The anthropolo- 
gist who learns a good deal about his neighbors in confidence must respect 
that confidence, except for overwhelming reasons, though it is of course 
conceivable that there might be occasion When he should not. No hard and 
fast rules can be laid down; these are matters of conscience rather than 
of science" (Beattie, 1955, p. 55). 

Beattie is absolutely correct in observing that no "hard and fast rules" 
can be prescribed, and as his own experience reveal s^ the problem assumes a 
complexity that goes beyond the question of what infonnatiDn should be re- 
vealed, to include problems related to both how it should be revealed; a di- 
rect report?, a suggestive hint?, and to whom should it be revealed? The 
problem is obviously mul ti-dimensibhal and many things should be considered 
if the decisions associated with the handling of potentially sensitive in- 
formation are to be more than perfunctory. 

My own analysis of this problem has led me to identify three factors 
which I think are important to consider in dealing With a particular bit of 
information, and in trying to decide what to do with it. I have labeled the 
three factors: utility, style, and context. The are defined and discussed 
as follows. 
Utility 

The concept of utility spans a contiuum from "definitely useful" to 
"definitely trivial." Obviously, one does not report everything one knows 
about a program* and for potentially volatile information one has to re- 
flect upon the implications for making certain knowledge public. Unfor- 
tunately* much of what the researcher discovers cannot be labeled definite- 
ly useful or definitely triv/ial ; thus, the decision must consider the in- 
foniiatidn as- it relates to the program as a whole, and' in particular, 
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its utility relative to the "description'' of the program to be derived 
from the research* Edward Shils preserves the complexity of the prob- 
lem, yet responds to it profoundly and precisely with the following state- 
ment: 

"••.particular truths must make concessions to privacy, as long as the 
concessions do not make inroads into the general truths discovered by 
the inquiry" (1959, p-154). 

Although this statement does not make the decision less difficult, it 
does concisely and accurately define the context within which the deci- 
sion should be made. Once a decision has been reached that certain in- 
formation IS useful and should be revealed, the second and third factors 
come into play. 

The style factor can be defined in terms of a continuum that ranges 
from "direct revelation" to "subtle implication." A simple illustration 
IS Beattie*s strategy, in the previous excerpt, of advising the medical 
authorities to be more careful with their drugs and paraphernalia. He 
decided the information was useful, but he did riot think it ethical to 
directly reveal what he knew. While studying ACSP I managed to reveal 
to the project director and to the coordinator that they held contrast- 
ing beliefs about a crucial aspect of the program by allowing them to 
"discover" one another's viewpoint in a meeting^ without me having to tell 
either one that they disagreed. The point is that sometimes orie consid- 
ers infonnation to be useful, yet simply stating it or reporting it 
straightforwardly may minimize the chances of it acutally being used and/ 
or may hurt some person or persons unnecessarily. 
Contexts 

The third factor, context, is also related to how information is 

23 

erJc 



revealed. For context, I have defined a continaam of "informal conversation" 
to "formal presentation." Like the previous two factors, this one is fairly 
easy to understand, but difficult to apply. It relates to the fact that the 
same piece of information can produce radically different results, depending 
Upon whether it appears in a final report, a memo, a meeting, or a private 
conversation. An example of the context in which information is revealed 
can be drawn from an experience I has as a member of a four-person team e~ 
valuating an alternative school. Approximately half-way through the first 
(and only) year of this program's existence, the director of the project, 
who was also the principal of the school, was transferred. His replacement 
was forced to divide his time between the school and another project he was 
directing. In addition ^ his philosophy as to how the school should be run 
differed drastically from his predecessor; and his style of management was 
more autocratic* whereas the former director's style was democratic. De- 
spite his sincere intentions, the new director created tremendous contro- 
versy and a number of the faculty members developed a personal dislike for 
hira. The evaluation team became party to much volatile information and we 
were aware that many problems were being created by the staff's attitude . 
toward the new director and his practices. At one point, the leader of 
bUr team was asked by some staff members to share their feelings and con- 
cerns with the director. The team leader met with the director and re- 
vealed to him* very frankly, much of what the staff felt and the problems 
it was. creating. Unfortunately^ the program was discontinued for reasons 
that went beyond the problems between the staff and th6 director. In our 
final report* however* we did not discuss the specific problems, conflicts, 
and accusations that revolved around the appointment of a new director, 
though we knew they were substantial; instead, v;e discussed, in general, 
the consequences that resulted from the change in directors. We felt that the 
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director needed to be apprised of the situation in detail, but in a final 
report written after the program's demise^ we felt it was sufficient to 
report only our summary conclusions as they pertained to the change in 
directors. 

I have attempted, in this discussion, to attempt to clarify the prob- 
lem of the ethical treatment of information by presenting certain concepts 
that I think are crucial to this issue. I do not think that the decisions 
concerning what to reveal and how to reveal it are necessarily r?ade any 
easier by considering the factors mentioned above, but I do xKnow from my 
own experience that frarriihg the problems in terms of these factors makes 
for more enlightened and conscientions decisions. 
final-StatemenL on -Handling Sensitive Information 

One final and overriding consideration which has been implicit in my 
discussion needs to be explicitly stated for the problems discussed above 
to be adequately treated. The primary consideration Under which my pre- 
vious discussion is subsumed is contained in the following statement by 
Howard Becker which, although specifically referring to the published re- 
port, is general izable to any uses of information by social scientists: 
"I assume that the scientist is not engaged in willful and malicious de- 
famation of character, that his published report has some reasonable 
scientific purpose, and therefore do not consider those cases in which a 
scientist might attempt, out of malice, ideological or personal, to de- 
stroy the reputation of persons or institutions. I further assume that 
the scientist is subject to no external constraint, other than that im- 
posed by his relationship to those he has studied, which would hinder 
him from reporting his results fully and freely" (1969, pp. 264-265). 
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Advant ages of the Ethnographic Approach 
the ethnographic approach to evaluation holds greater promise for 
the disci pi ine than does the experimental approach which characterizes 
much of the evaluation work being done. The reasons for this conclu- 
sion have to do primarily with: (1) the focus the experimental model 
brings to bear upon the research setting, (2) the concept of the "pro- 
gram" created;by the experimental model, and (3) the lack of reciprocity 
inherent to experimental research. The specific nature of each of these 
observations is discussed below. 

The focus in the experimental model is on particular variables and 
the statistical relatfonsMps between them. The meaning of events in this 
model is taken to be self-evident. The particular variables under inves- 
tigation are assumed to be discrete, self-contained, essentially static 
in form; each has singular meaning. In this model, understanding means 
attempting to discover causal relationships between entities whose mean- 
ings and essential form do not change during the course of the observa- 
tion. This kind of "explanation" can be likened to the assumption as to 
"the nature of comprehension" discussed in an article entitled "Pre-re- 
quisites for Understanding: Implications for the Design of instructional 
Strategies and Materials," By Buford Wilson. He states: "Typically un- 
derstanding what another person says seems to be a smooth effortless, au- 
tomatic process. People speak and we just know what they mean. Partly 
as a result of this ease of understanding we are led to false conclusions 
about the way that words work in allowing us to understand other people 
and written messages. It seeras natural to assume that we learn words arid 
their meanings, and that when we later hear the words their meanings are 
evoked. Understanding, then, is just being aware of the meanings of words. 
Words function in lariguage, the story goes, as symbols or representatioris 
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of meanings. The word is heard or seen, Its meaning is evoked, and under- 
standing occurs" (1978, p. 17). 
However, he continues: 

"Words do not function as referents' Vf objects or meanings, rather words 
act as cues on the basis of which we construct meaningful interpretations 
of messages or interactions. For example, the words "How are you?" can 
mean a wide variety of things depending upon how they are said and the 
context in which they occur. ..the meaning of the phrase is inferred bh 
the basis of the listener's knowledge about what is happening betv/een the 
speaker and himself and how this might relate to other interactions, or 
special knowledge he has about himself, the speaker, or their relation- 
ship. The words do not evoke a given set of meanings. If you play the 
game of changing contexts, changing relations between people, and ehang-- 
ing the demands of the situation, you can create a seemingly endless set 
of meanings for the phrase 'how are you'" (Ibid, p. 17). 

Wilson's point is pertinent to evaluation studies. Specific actions, 
events, "treatments," and "variables" are the "words" in a given program, 
and attempts to understand them as discrete and denotative are misguided. 
Webster's Dictionary is not Melville's Moby Pick , and though the primary 
elements of both are the same, it is absurd to think that the immensely 
evocative imagery of the latter could be discovered in the former, tike- 
wise, for research focused upon educational and other social action pro- 
grams to be worthwhile, the facade that is self-evident in the research 
setting must be penetrated if the true nature of the phehomehoh is to be 
revealed. The perspective assumed by the experimental paradip is that 
there no meaning beyond denotation. The ethnographic approach, as pre- 
sented in this paper, is designed to go beyond the denotative, to pene- 
trate the facade in search of meaning. 



Evaluators adhering to the experimental model of research must focus 
upon discrete events or phenomena and assume each has meaning only in a 
strictly defined, self-contained, denotative sense. The problem with con- 
ceiving of a program (or any research setting) in these terms is that the 
concept of the program created by the researcher fails to capture the es- 
sense of the program. This point can best be understood in light of 
George Kelly's explanation of the pf^oeess of human thought and 
its implications for psychological research. In his book, A Theory of 
Personality: The Psychology of Pe ^nal ednstructs , Kelly argues that in 
attempting to understand the subtleties of thought, one must realize that 
jneaning for an individual is created through constructs that are dichbto- 
mous in nature. Traditionally, mental constructs were viewed as being 
uni-dimensional , and Ke11y argues that this results in an inadequate diag- 
nosis because the client's entire construct is not revealed. He states: 
''Consider a person's use of the construct of Tespeoi vs. Contempt. Under 
conventional logic one would consider these as two separate concepts. If 
we wished to understand the person's use of the term " respect," we might 
seek to find out how broadly he applied the term, how he "generalized the 
concept." We would want to know what acts he considered to be characte- 
rized by "respect" and what acts he did not consider "respectful." Thus, 
we might be able to discover By the method of varying concomitants just 
what abstraction among the acts fie had been able to make. But when we 
approach the thinkiiig of a person, say a clinic client, in this way^ we 
roiss a great deal. We vrCns it Because we are iaoiily dssimihg that ev^ 
erythtng which he does not construe as ''respect" is irrelevant. Yet his 
use of the construct may be particularly meaningful because of what he 
excludes rather than because of what he includes" (1963, pp.70-71). 
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Kelly goes on to explain that in order to understand what "respect" 
means to a given individual, one must go beyond a strict interpretation 
and explore what the person "sees as relevantly opposed to respect" (Ibid, 
p. 71). In summary, Kelly states: 

"The psychologist who employs the approach of the psychology of personal 
constructs is led always to look for the contrasting elements of his cli- 
ent's constructs as well as the similar elements. Until he has some no- 
tion of the contrast, he does not presume to understand the similarity" 
(Ibid, p. 71). 

The experimental model suffers because it is preoccupied with the 
"similar elements," and it treats "contrasting elements" as irrelevant. 
The meaning of a construct for an individual is derived from the inter- 
play between the contrasting elements. Meaning for the program is cre- 
ated by the interplay between sirailar arid constrasting elements^ within 
the realm of both the defactb and "extended" programs, the concept of 
a program, as a strictly defined set of discrete events ^ can accurately 
reflect neither the interaction that takes place between the many ele- 
ments of a program, nor can it accommodate the meaning created by the 
interplay of elements. The ethnographic evaluator, like the psycholo- 
gist using the "psychology of personal constructs" applies an expanded 
definition of what constitutes the relevant elements iri the investiga- 
tion. For example, I studied the arguments for arid against programs 
like ACSP that increase the discretionary options of the court* arid 
briefly presented both sides as part of my evaluation; I do not think 
the typical experimental study would have explored this particular issue. 

For the discipline of evaluation, the precedirig discussidri has fur- 
ther implications. The evaluator, like the anthropologist, cannot prac- 
tice his trade without the support of the populace iri the research 
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setting. It is inunaterial whether brie is studying ah alternative schddl 
or an aboriginal tribe; if one does not gairi the right to be present then 
very little, if any, data can be collected. Any research conducted in 
the "field" is dependent upon the cooperation of the people involved. In 
the case of the traditional research paradigm, "cooperation" frequently 
involves a willingness to submit to control and mariipulatibri. Ironically, 
the experimental model spawns evaluation studies whose success is depen- 
dent upon the total compliance of the people in the research setting. 
But, it is seldom that the research which requires complete eobperatiori 
proves to be meariirigful to anyone other than the researcher arid his or 
her colleagues. The basic problem is that evaluation studies conducted 
within the experimental paradigm do not, by their nature, enable a re- 
ciprocal relationship to be established between the researcher and the 
people involved in the study. 

Marshall Sahlins, in an article erititled "Ori the Sociology of Prim- 
itive Exchange" (1969), discusses the role that reciprocal exchange plays 
in the "primitive" society. Different types of excharige symbolize dif- 
ferent kinds of relationships between people and groups. He specifies 
three points on a coritiriuum of reciprocal exchange: 0) "Generatized re- 
oiprooits refers to trarisactioris that are putatively altruistic, trans- 
actions on the line of assistance given, arid if possible arid riecessary, 
assistarice returned"; (2) "Baica^oed veoipvooiiy refers to direct exchange. 
In precise balarice, the reciprocation is the customary equivalerit of the 
thirig received arid is without delay", (3) "negative reaiprociiy is the 
attempt to get somethirig for riothirig with impuriity..." (.pp. 147-148) . 

The relationship between the evaluator arid the subjects of the eval- 
uation has been characterized by "riegative reciprocity." The evaluator 
is nianipulative, demaridirigi arid disruptive through attempts to mairitairi 



EKLC 



30 



the consistency of treatment and conditions. Yet when the study is com- 
pleted arid the evaluatdr pulls outj a report is written which is usually 
of little value to the people who must remain with the program. This is 
becasue it is generally Uritimely (discussirig events of the past which are 
like "water over the dam")^ and iricdmpreherisible ("reject the null", "sig- 
nificant at the .05 level", etc.). The evaluator indeed gets "something 
for nothing^" and the people who cooperated are left to scratch their 
heads and wonder as to the ineanirig of it all. 

For evaluatiori to be meariirigful, the relationship between parties 
to the study must be characterized by a more "balanced reciprocity;" not 
in the strict sense of immediate reciprocation, but in the sense of ari 
interactive relatibriship that is symbolized by a cdntiriUous exchange of 
relevant informatibri. The ethnographic approach strives to create a re- 
lationship based on "balanced reciprocity." Furthermore, as information 
is exchanged it is done so iri terms of a symbol system that all parties 
can understarid. 

Iri summary^ the theme Underlying the criticisms pertairiirig to: fo- 
cus ^ cdricept df prdgram, arid lack df reciprdcity is that the e>:perimerit- 
al apprdach to evalUatidri is primarily cdncerried with payirig hdmage td a 
scientific traditidri, the result of Which is dfteri a disruptive ritual- 
ized process that cdritributes little td either the advaricement of gener- 
al kridwledge dr td kndWledge about a specific prdgrami whereas evalUa- 
tidri ddrie from the ethriographic perspective attempts td follow a cdUrse 
suggested by the Unique circumstances arid needs df the particular pro- 
gram being studied^ using this lead to produce an interpretive analysis 
df the prdgram. 
Concl Usidri 

It has been the intent df this paper td present, in incipient form^ 
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a model for conducting program evaltiatibris frbtii an ethnographic perspec- 
tive, it has been argued that this model has advantages over the exper- 
imental model that makes it more relevant to the needs Of both the prac- 
titioners and Clients of evaldatibri; Finally, it is the belief of this 
author that the ethnographic approach, as it is refined on the basis of 
practical application, offers the greatest potential - cohceputally and 
methodologically - for investigating educational and Social action pro- 
grams* improving them, and ascertaining their impact. 
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._ FIGURE 1 , : 

Model For Ah Ft.hnograjjhic Approach to Program Evaloatidh 
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